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(57) A game state manager (201) manages the 
state of an AR game (information that pertains to ren- 
dering of each virtual object (102), the score of a player 
(101), the AR game round count, and the like). An ob- 
jective viewpoint video generator (202) generates a vid- 
eo of each virtual object (102) viewed from a camera 
(103). An objective viewpoint video composition unit 



(203) generates a composite video of the video of the 
virtual object (102) and an actually sensed video, and 
outputs it to a display (1 06) . A subjective viewpoint video 
generator (212) generates a video of the virtual object 
(1 02) viewed from an HMD (1 07). A subjective viewpoint 
video composition unit (213) generates a composite vid- 
eo of the video of the virtual object (1 02) and an actually 
sensed video, and outputs it to the HMD (1 07). 
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eos of the real space and the virtual object viewed 

from said player's viewpoint position; 

and 

the display means for displaying to the player the 
augmented reality video viewed from said player's s 
viewpoint position. 

[001 5] The augmented reality presentation apparatus 
as a preferred embodiment of the present invention fur- 
ther comprises the following feature described in claim 10 

^3. 

[0016] That is, said augmented reality presentation 
means further comprises: 

the second video generation means for generating is 
a video of the virtual object viewed from said play- 
er's viewpoint position; 
and 

the display means for displaying to the player the 
video of the virtual object viewed from said player's 20 
viewpoint position on a display surface through 
which the player can visually see the real space. 

[001 7] The augmented reality presentation apparatus 
as a preferred embodiment of the present invention fur- 25 
ther comprises the following feature described in claim 
4. 

[0018] That is, information generation means for gen- 
erating information that pertains to rendering of the vir- 
tual object, and 30 

in that said first video generation means and said 
second video generation means generate videos of the 
virtual object using the information that pertains to ren- 
dering of the virtual object. 

[001 9] The augmented reality presentation apparatus 35 
as a preferred embodiment of the present invention fur- 
ther comprises the following feature described in claim 
5. 

[0020] That is, said information generation means 
generates information of an outer appearance of the vir- 40 
tual object and information of a position/posture of the 
virtual object as the information that pertains to render- 
ing of the virtual object. 

[0021 ] The augmented reality presentation apparatus 
as a preferred embodiment of the present invention fur- 45 
ther comprises the following feature described in claim 
6. 

[0022] That is, parameters of said first video sensing 
means are known, and 

said first video generation means generates the so 
video of the virtual object viewed from said first view- 
point position in accordance with the known parameters. 
[0023] The augmented reality presentation apparatus 
as a preferred embodiment of the present invention fur- 
ther comprises the following feature described in-claim 55 
7. 

[0024] That is, some of parameters of said first video 
sensing means are variable, 



said apparatus further comprises measurement 
means for measuring changes of the parameters, 
and 

said first video generation means generates.the vid- 
eo of the virtual object viewed from said first view- 
point position in accordance with the parameters 
measured by said measurement means. 

[0025] When the parameters of the objective view- 
point video sensing means, the objective viewpoint vid- 
eo generation means receives parameters from the 
measurement means, and generates an objective view- 
point video according to the received parameters. 
[0026] The augmented reality presentation apparatus 
as a preferred embodiment of the present invention fur- 
ther comprises the following feature described in claim 
8. 

[0027] That is, the parameters of said first video sens- 
ing means measured by said measurement means in- 
clude at least one of a viewpoint position/posture, and 
zoom ratio. 

[0028] The objective viewpoint video of a virtual object 
is generated in accordance with camera parameters 
(external parameters (viewpoint position/posture) and 
internal parameters (zoom ratio, aspect ratio, optical ax- 
is central position, distortion ratio)) of the objective view- 
point video sensing means. The camera parameters 
measured by the measurement means preferably in- 
clude all parameters to be changed dynamically of those 
parameters. 

[0029] Other features and advantages of the present 
invention will be apparent from the following description 
taken in conjunction with the accompanying drawings, 
in which like reference characters designate the same 
or similar parts throughout the figures thereof. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0030] The accompanying drawings, which are incor- 
porated in and constitute a part of the specification, il- 
lustrate embodiments of the invention and, together with 
the description, serve to explain the principles of the in- 
vention. 

Fig. 1 is a view showing the first embodiment; 
Fig. 2 is a block diagram showing the flow of proc- 
esses of the first embodiment; 
Fig. 3 is a diagram showing the generation process 
of a coordinate conversion matrix used to convert 
the object coordinate position of a virtual object into 
an image coordinate position viewed from a given 
viewpoint; 

Fig. 4 is a block diagram showing the flow of proc- 
esses of the third embodiment; 
Fig. 5 is a view showing a video presented to a play- 
er in the first to fourth embodiments; 
Fig. 6 is a view showing a video presented to a play- 
er in the fifth embodiment; 
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AR game (information that pertains to rendering of the 
virtual objects 1 02, the score of the player 1 01 , AR game 
round count, and the like). Note that the information that 
pertains to rendering of the virtual objects 1 02 includes 
information of the outer appearance of each virtual ob- 
ject 102, and information of the position/posture of each 
virtual object 102 in the world coordinate system. 
[0043] The information of the outer appearance of 
each virtual object 1 02 is information that pertains to pol- 
ygons which build the virtual object 1 02, i.e., the number 
of polygons, the coordinate values of polygons, the 
colors of polygons, and the like. When each virtual ob- 
ject 102 has undergone texture mapping, the informa- 
tion of the outer appearance of each virtual object 102 
includes a texture size, texture file name, and the like. 
[0044] Reference numeral 204 denotes a camera pa- 
rameter measurement/management unit which meas- 
ures/manages camera parameters as parameters of the 
HMD 107 and camera 103. The camera parameters to 
be managed by the camera parameter measurement/ 
management unit 204 includes viewpoint position/pos- 
ture information as external parameters, and informa- 
tion of the field angle, focal length, distortion, and the 
like as internal parameters. The camera parameter 
measurement/management unit 204 stores the internal 
parameters of the HMD 107 as known information. At 
the same time, the unit 204 measures the external pa- 
rameters (information of the viewpoint position and pos- 
ture) of the HM D 1 07, and manages the camera param- 
eters of the HMD 1 07. Furthermore, the camera param- 
eter measurement/management unit 204 manages 
camera parameters of the camera 103 as known infor- 
mation. 

[0045] Reference numeral 202 denotes an objective 
viewpoint video generator built in the AR game appara- 
tus 104. The objective viewpoint video generator 202 
generates objective viewpoint virtual videos on the basis 
of the information that pertains to rendering of the virtual 
objects 1 02, which is input from the game state manager 
201. 

[0046] Reference numeral 203 denotes an objective 
viewpoint video composition unit built in the AR game 
apparatus 104, which generates an objective viewpoint 
AR video by compositing objective viewpoint virtual vid- 
eos generated by the objective viewpoint video gener- 
ator 202, and an objective viewpoint actually sensed vid- 
eo input from the camera 103. 

[0047] Generation of the subjective viewpoint video 
will be explained below. 

[0048] The game state manager 201 updates the in- 
formation that pertains to rendering of the virtual objects 
1 02 as needed, and outputs the updated information to 
a subjective viewpoint video generator 21 2. The subjec- 
tive viewpoint video generator 212 generates videos of 
the virtual objects 1 02 (subjective viewpoint virtual vid- 
eo) viewed from the viewpoint position/posture of the 
HMD107, on the basis of the information that pertains 
to rendering of the virtual objects 102, which is input 



from the game state manager 201 , and the camera pa- 
rameters of the HMD 1 07, which are input from the cam- 
era parameter measurement/management unit 204. 
[0049] The generated subjective viewpoint virtual vid- 
5 eo is output to a subjective viewpoint video composition 
unit 213. 

[0050] The subjective viewpoint video composition 
unit 21 3 receives from the HMD 1 07 the subjective view- 
point actually sensed video that the player 1 01 watches 

w via the HMD 107. The subjective viewpoint video com- 
position unit 213 generates a subjective viewpoint AR 
video as a composite video of this subjective viewpoint 
actually sensed video, and the subjective viewpoint vir- 
tual videos input from the subjective viewpoint video 

is generator 212, and outputs that subjective viewpoint AR 
video to the HMD 1 07. The subjective viewpoint AR vid- 
eo is displayed on the display screen 501 of the HMD 
107, and the player 101 plays the AR game while re- 
viewing this subjective viewpoint AR video. 

20 [0051] Generation of an objective viewpoint video will 
be explained below. 

[0052] The game state manager 201 updates the in- 
formation that pertains to rendering of the virtual objects 
102 as needed, and outputs the updated information to 

25 the objective viewpoint video generator 202. ■ 

[0053] Since generation of the videos of the virtual ob- 
jects 1 02 is implemented by the same processes as 
those in generation of the subjective viewpoint video, a 
detailed description thereof will be omitted. ] 

30 [0054] The generated objective viewpoint virtual vid- 
eo is output to the objective viewpoint video composition 
unit 203. 

[0055] The objective viewpoint video composition unit 
203 receives an objective viewpoint actually sensed vid- 

35 eo from the camera 1 03. The objective viewpoint video 
composition unit 203 generates an objective viewpoint 
AR video as a composite video of this objective view- 
point actually sensed video, and the objective viewpoint 
virtual videos input from the objective viewpoint video 

40 generator 202, and outputs the objective viewpoint AR 
video to the display 106. This objective viewpoint AR 
video is displayed on the display 106, and the watcher 
can see the overall view of the AR game by reviewing 
this objective viewpoint AR video and can recognize the 

45 current situation of the AR game. 

[0056] The image coordinate values of the vertices of 
the polygons that build each virtual object 102 viewed 
from each viewpoint (HMD 107 or camera 103) can be 
computed using the coordinate conversion matrix Ml or 

50 M2 from the object coordinate system into the image co- 
ordinate system. The generation process of such coor- 
dinate conversion matrix will be explained below using 
the block diagram shown in Fig. 3. 
[0057] A coordinate conversion matrix Mm from the 

55 object coordinate system into a world coordinate system 
is computed on the basis of the information of the posi- 
tion/posture of each virtual object 102. Also, coordinate 
conversion matrices Mcl and Mc2 from the world coor- 
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program code, an objective viewpoint AR video can be 
presented to a third party other than the player 1 01 in 
the AR game using the video see-through HMD 107. 

[Second Embodiment] 

[0076] In the first embodiment, the HMD 1 07 is of vid- 
eo see-through type. However, if the HMD 107 is of op- 
tical see-through type, the player 101 can still play the 
AR game. 

[0077] Fig. 13 shows an optical see-through HMD 
1301. Note that Fig. 13 schematically illustrates the 
HMD 1301, and the present invention is not limited to 
the size and shape shown in Fig. 13. 
[0078] Reference numeral 1301 denotes an optical 
see-through HMD; and 1302, the eye of the player 101 . 
[0079] On the display screen 501, only videos of the 
virtual objects 102 (subjective viewpoint virtual videos) 
are displayed. On the other hand, a video of a real space 
is seen behind the display screen 501 when viewed from 
the position of the eye 1302. Hence, the player can re- 
view videos of the virtual objects 1 02 and the real space 
seen behind the display screen 501 when viewed from 
the position of the eye 1302 to overlap each other by 
observing the display screen 501 . 
[0080] Fig. 14 is a block diagram showing the flow of 
processes of this embodiment. 

[0081] A video output from the AR game apparatus 
1 04 to the HMD 1 301 is a subjective viewpoint virtual 
video alone, as described above. Hence, the HMD 1301 
and subjective viewpoint video generator 212 are elec- 
trically connected via a cable, and a subjective viewpoint 
virtual video which is generated by the subjective view- 
point video generator 212 on the basis of the position 
and posture of the HMD 1 301 is sent from the subjective 
viewpoint video generator 212 to the HMD 1301. Note 
that this embodiment has no subjective viewpoint video 
composition unit 21 3 in the first embodiment. 
[0082] Also, the method of generating an objective 
viewpoint AR video is the same as that which has been 
explained in the first embodiment. 
[0083] The internal block diagram of the AR game ap- 
paratus 104 in this embodiment is substantially the 
same as that shown in Fig. 7, except that the objective 
viewpoint video composition unit 213 is removed from 
the arrangement shown in Fig. 7. 
[0084] In the flow chart of this embodiment, steps 
S804, S805, and S806 in Fig. 8 are rewritten as follows. 
More specifically, upon generating a subjective view- 
point AR video (only videos of the virtual objects 102 
based on the position and posture of the HMD 1301 in 
this embodiment) to be displayed on the HMD 1301, 
steps S804 and S805 are skipped. That is, in step S804 
the objective viewpoint video composition unit 203 re- 
ceives an actually sensed video sensed by the camera 
103. In step S805, the objective viewpoint video com- 
position unit 203 generates an objective viewpoint AR 
video. In step S806, the objective viewpoint video com- 



position unit 203 outputs the objective viewpoint AR vid- 
eo to the display 1 06, and the subjective viewpoint video 
generator 21 2 outputs a subjective viewpoint virtual vid- 
eo to the HMD 1301. 

5 [0085] The flow chart obtained by modifying the con- 
tents of Fig. 8 as described above is that in this embod- 
iment, and this embodiment is controlled by a program 
code according to this modified flow chart. 
[0086] With the aforementioned arrangement of the 

10 apparatus, augmented reality presentation method, and 
program code, an objective viewpoint AR video can be 
presented to a third party other than the player 1 01 in 
the AR game using the optical see-through HMD 1301 . 

15 [Third Embodiment] 

[0087] In the first and second embodiments, the cam- 
era parameters of the camera 1 03 are fixed. That is, an 
objective viewpoint AR video is generated based on the 
20 camera parameters of the camera 1 03 which is fixed in 
position. The camera parameter data of the camera 103 
are stored as permanent values in the RAM 703 of the 
AR game apparatus 104. 

[0088] A case will be examined below wherein the 
25 viewpoint position, posture, and zooming ratio t>f the 
camera 103 are changed in real time to those that the 
player or a third party other than the player 101 (watcher 
or operator) wants. That is, a case will be examined be- 
low wherein the camera parameters of the camera 103 
30 are changed in real time. Note that this embodiment us- 
es a video see-through HMD as in the first embodiment. 
However, the HMD that can be used in this embodiment 
is not limited to the video see-through type, but an opti- 
cal see-through HMD may be used, as can be seen from 
35 the description of the second embodiment and this em- 
bodiment. 

[0089] When the position, posture, and zooming ratio 
of the camera 103 are to be changed in real time, a 
measurement means as a means for measuring the po- 

40 sition, posture, and zooming ratio of the camera 103 
must be added to the first embodiment. 
[0090] Fig. 4 is a block diagram showing the flow of 
processes in this embodiment. The flow of processes in 
this embodiment will be described below using Fig. 4. 

45 [0091] Like in the first embodiment, in this embodi- 
ment the camera parameter measurement/manage- 
ment unit 204 measures and manages the camera pa- 
rameters of the HMD 107, and holds some internal pa- 
rameters of the camera 103 as known information. Un- 

50 like in the first embodiment, in this embodiment the cam- 
era parameter measurement/management unit 204 
controls a sensor (not shown) attached to the camera 
103 to measure the position, posture, and zooming ratio 
of the camera 1 03. 

55 [0092] When the player 101 has changed the position, 
posture, and zooming ratio of the camera 1 03 via an in- 
terface (not shown), the camera parameter measure- 
ment/management unit 204 m asures the position, pos- 
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augmented reality presentation method, and program 
code can be applied. In this case, such application can 
be implemented by connecting the camera system of 
this embodiment to the AR game apparatus 1 04 in place 
of the camera 103 in the third embodiment. 
[0110] With the aforementioned arrangement of the 
apparatus, augmented reality presentation method, and 
program code, a plurality of objective viewpoint AR vid- 
eos sensed by a plurality of cameras can be displayed 
on the display 106. 

[Fifth Embodiment] 

[0111] In the first to fourth embodiments, an objective 
viewpoint AR video is presented to a third party other 
than the player via the display 1 06 by outputting it to the 
display 106. Alternatively, the objective viewpoint AR 
video may be presented to the player 101. That is, a 
display area 601 shown in Fig. 6 is assured on the dis- 
play screen 501 of the HMD (which can be of either vid- 
eo or optical see-through type) that the player 1 01 
wears, and the objective viewpoint AR video is dis- 
played there. In Fig. 6, a video displayed on this display 
screen 501 will be referred to as an augmented video 
hereinafter. 

[0112] In order to generate this augmented video, a 
program code for setting the display area 601 on the 
display screen 501 , and writing an objective viewpoint 
AR video on that display area 601 is stored in the RAM 
703 of the AR game apparatus 104 in addition to the 
program code according to the flow chart shown in Fig. 
8. As a result, by executing this program code, the aug- 
mented video can be displayed on the display screen 
501. 

[0113] Fig. 11 is a flow chart of the program code for 
writing the objective viewpoint AR video on the display 
area 601 mentioned above. 

[0114] It is checked in step S1101 if the display area 
601 is assured. Selection as to whether or not the dis- 
play area 601 is assured can be implemented by pro- 
viding this selection switch to an operation device (not 
shown) used when the player 101 plays the AR game. 
Or such selection can be implemented by inputting a 
command indicating whether or not the display area 601 
is assured from the console 705. 
[0115] In step S1102, the display position of the dis- 
play area 601 is input. The display position is input from 
the console 705. Alternatively, the player 101 may input 
the display position using the aforementioned operation 
device. 

[0116] In step S1103, the size of the display area 601 
is input. The size is input from the console 705. Alterna- 
tively, the player 101 may input the size using the afore- 
mentioned operation device. 

[01 17] In step S1 1 04, the display area 601 , the setups 
of which have been determined in steps S1102 and 
S1103, is assured on the display screen 501 . 
[01 18] In step S1 1 05, the objective viewpoint AR vid- 



eo generated by the objective viewpoint video compo- 
sition unit 203 is rendered on the display area 601 . As 
a consequence, the augmented video can be generat- 
ed. 

5 [0119] Note that this augmented video may be output 
to the display 106. 

[0120] Selection of whetherthe display area 601 is set 
and the display position and size of the display area 601 
may be automatically set/changed not by the player but 
10 in accordance with the progress of the game. In this 
case, the game state manager 201 determines these 
parameters. 

[0121] The display area 601 may be the entire area 
of the display screen 501 . 
15 [0122] With the aforementioned arrangement of the 
apparatus, augmented reality presentation method, and 
program code, both the subjective and objective view- 
point AR videos can be presented to the player. 

20 [Sixth Embodiment] 

[0123] A plurality of players 101 may join the AR 
game. In this case, subjective viewpoint AR videos from 
individual subjective viewpoints must be provided to the 

25 individual players. Fig. 15 shows the internal arrange- 
ment of the AR game apparatus for providing subjective 
viewpoint AR videos from the players* subjective view- 
points to the individual players. Note that the camera 
103 is fixed in position, and the processes that peVtain 

30 to the camera 1 03 and display are the same as those in 
the first embodiment. 

[0124] Fig. 15 shows the internal arrangement for 
three players. Hence, the AR game apparatus 1 04 com- 
prises HMDs 107A, 107B, and 107C, subjective view- 

35 point video composition units 21 3A, 21 3B, and 21 3C, 
and subjective viewpoint video generators 21 2A, 21 2B, 
and 21 2C in correspondence with three players a, b, and 
c. Three players a, b, and c respectively wear the HMDs 
107A, 107B, and 107C. This embodiment uses a video 

40 see-through HMD, but an optical see-through HMD may 
be used. In this case, the subjective viewpoint video 
composition units for the three HMDs can be omitted. 
[0125] When the AR game starts, the HMDs, subjec- 
tive viewpoint video generators, and subjective view- 

45 point video composition units execute the same proc- 
esses as those described in the first embodiment, and 
subjective viewpoint AR videos generated for the indi- 
vidual players are output to the HMDs 1 07A, 107B : and 
107C that the players wear. 

so [0126] Note that the aforementioned arrangement of 
the apparatus and augmented reality presentation 
method are not limited to three players, as is obvious 
from the above description. 

[0127] With the aforementioned arrangement of the 
55 apparatus and augmented reality presentation method, 
subjective viewpoint AR videos can be provided to a plu- 
rality of players. 
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viewpoint positions, objective viewpoint AR videos 
viewed from the same viewpoint at different times, or 
subjective viewpoint A R videos of the respective players 
on one paper sheet. In this case, the print controller301 
selects a viewpoint in accordance with a command input 
from the console 705 and stores the AR video viewed 
from the selected viewpoint. The print controller 301 de- 
termines a layout for printing a plurality of stored images 
on one paper sheet in accordance with a command input 
from the console 705 and outputs them to the printer 
302. 

[0142] The command may be automatically input to 
the print controller 301 in accordance with the progress 
of a game without using the console 705. In this case, 
a game state manager 201 transmits a command to the 
print controller 301 . The command may be transmitted 
at a fixed timing preset in accordance with the progress 
of the game or a dynamic timing corresponding to the 
progress of the game such that the distance between 
the player and a virtual object 1 02 becomes a predeter- 
mined interval or less. 

[01 43] The camera 1 03 (or camera 901 ) can be locat- 
ed at an arbitrary position. The third party can easily 
grasp the state of the game when the camera is located 
at a position where the entire AR space can be observed 
from the bird-view position or a position where the upper 
half image (so-called bust shot) of the player can be 
sensed from the front. 

[0144] As described above, according to the present 
invention, a video of the AR game viewed from the sub- 
jective viewpoint of the player can be generated, and, 
simultaneously, a video of the AR game viewed from an 
objective viewpoint to see the overall view of the game 
field or a video of the AR game viewed from a viewpoint 
the watcher wants can be generated. 
[0145] As many apparently widely different embodi- 
ments of the present invention can be made without de- 
parting from the spirit and scope thereof, it is to be un- 
derstood that the invention is not limited to the specific 
embodiments thereof except as defined in the append- 
ed claims. 

[0146] A game state manager (201) manages the 
state of an AR game (information that pertains to ren- 
dering of each virtual object (102), the score of a player 

(101) , the AR game round count, and the like). An ob- 
jective viewpoint video generator (202) generates a vid- 
eo of each virtual object (102) viewed from a camera 
(103). An objective viewpoint video composition unit 
(203) generates a composite video of the video of the 
virtual object (102) and an actually sensed video, and 
outputs it to a display (1 06). A subjective viewpoint video 
generator (212) generates a video of the virtual object 

(1 02) viewed from an HMD (1 07). A subjective viewpoint 
video composition unit (213) generates a composite vid- 
eo of the video of the virtual object (102) and an actually 
sensed video, and outputs it to the HMD (107). 



20 
Claims 

1. An augmented reality presentation apparatus for 
superimposing a virtual object in a real space, char- 

5 acterized by comprising: 

augmented reality presentation means for su- 
perimposing the virtual object viewed from a 
player's viewpoint position in the real space 
*0 viewed from said player's viewpoint position; 

the first video sensing means for sensing a vid- 
eo of the real space viewed from a first view- 
point position which differ from said player's 
viewpoint position; 

the first video generation means for generating 
a video of the virtual object viewed from said 
first viewpoint position; 
and 

the first video composition means for compos- 
iting an augmented reality video viewed from 
said first viewpoint position on the basts of said 
videos of the real space and the virtual object 
viewed from said first viewpoint position. 

2. The apparatus according to claim 1 , characterized 
in that said augmented reality presentation means 
further comprises: 



25 



the second video sensing means for sensing a 
30 video of the real space viewed from said play- 

er's viewpoint position; 

the second video generation means for gener- 
ating a video of the virtual object viewed from 
said player's viewpoint position; 

35 the second video composition means for com- 

positing an augmented reality video viewed 
from said player's viewpoint position on the ba- 
sis of said videos of the real space and the vir- 
tual object viewed from said player's viewpoint 

40 position; 
and 

the display means for displaying to the player 
the augmented reality video viewed from said 
player's viewpoint position. 

45 

3. The apparatus according to claim 1 , characterized 
in that said augmented reality presentation means 
further comprises: 

50 the second video generation means for gener- 

ating a video of the virtual object viewed from 
said player's viewpoint position; 
and 

the display means for displaying to the player 
55 the video of the virtual object viewed from said 

player's viewpoint position on a display surface 
through which the player can visually see the 
real space. 
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through which the player can visually see the 
real space. 

13. The method according to any one of claim 10 to 12, 
characterized by further comprising the information 
generation step of generating information that per- 
tains to rendering of the virtual object, 

and 

in that in said first video generation step and 
said second video generation step, videos of the vir- 
tual object are generated using the information that 
pertains to rendering of the virtual object. 

14. The method according to claim 13, characterized in 
that said information generation step includes the 
step of generating information of an outer appear- 
ance of the virtual object and information of a posi- 
tion/posture of the virtual object as the information 
that pertains to rendering of the virtual object. 

15. The method according to claim 10 or 13, character- 
ized in that parameters of means for sensing said 
first viewpoint video are known, and 

said first video generation step includes the 
step of generating the video of the virtual object 
viewed from said first viewpoint position in accord- 
ance with the known parameters. 

16. The method according to claim 1 0 or 13, 

characterized 
in that some of parameters of means for sensing a 
video viewed from said first viewpoint position are 
variable, 

said method further comprises the measure- 
ment step of measuring changes of the param- 
eters, 
and 

said first video generation step includes the 
step of generating the video of the virtual object 
viewed from said first viewpoint position in ac- 
cordance with the parameters measured in the 
measurement step. 

17. The method according to claim 1 6, characterized in 
that the parameters of the means for sensing a vid- 
eo viewed from said first viewpoint position meas- 
ured in the measurement step include at least one 
of a viewpoint position/posture, and zoom ratio. 

18. The method according to claim 1 0, characterized in 
that when a plurality of means for sensing a video 
viewed from said first viewpoint position are 
present, 

said method further comprises the selection 
step of receiving a plurality of videos of the real 
space viewed from a first viewpoint position 



from the plurality of means for sensing a video 
viewed from said first viewpoint position, and 
outputting the video of the real space viewed 
from a first viewpoint position input from one se- 

5 lected means for sensing a video of said first 

viewpoint position to means for compositing a 
first viewpoint video, and 
said first video composition step includes the 
step of generating a video of the virtual object 

io viewed from said first viewpoint position using 

parameters of the means for sensing a video 
viewed from a first viewpoint position selected 
in the selection step. 

is 19. A storage medium storing a program code for su- 
perimposing a virtual object in a real space when 
said program code is loaded by a computer, 
characterized by comprising: 

20 a program code of the augmented reality pres- 

entation step of superimposing the virtual ob- 
ject viewed from a player's viewpoint position 
in the real space viewed from said player* view- 
point position; 

25 a program code of the first video sensing step 

of sensing a video of the real space viewed from 
a first viewpoint position which differ from said 
player's viewpoint position; 
a program code of the first video generation 
30 step of generating a video of the virtual object 

viewed from said first viewpoint position; 
and 

a program code of the first video composition 
step of compositing an augmented reality video 
35 viewed from said first viewpoint position on the 

basis of said videos of -the real space and the 
virtual object viewed from said first viewpoint 
position. 

*o 20. The medium according to claim 19, characterized 
in that the program code of the augmented reality 
presentation step further comprises: 

a program code of the second video sensing 
45 step of sensing a video of the real space viewed 

from said player's viewpoint position; 
a program code of the second video generation 
step of generating a video of the virtual object 
viewed from said player's viewpoint position; 
50 a program code of the second video composi- 

tion step of compositing an augmented reality 
video viewed from said player's viewpoint posi- 
tion on the basis of said videos of the real space 
and the virtual object viewed from said player's 
55 viewpoint position; 

and 

a program code of the display step of displaying 
to the player the augmented reality video 
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